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Twitter, a popular social media outlet, has evolved into a vast source of linguistic data, rich 
with opinion, sentiment, and discussion. Due to the increasing popularity of Twitter, its perceived 
potential for exerting social influence has led to the rise of a diverse community of automatons, 
commonly referred to as bots. These inorganic and semi-organic Twitter entities can range from the 
benevolent (e.g., weather-update bots, help-wanted-alert bots) to the malevolent (e.g., spamming 
messages, advertisements, or radical opinions). Existing detection algorithms typically leverage 
metadata (time between tweets, number of followers, etc.) to identify robotic accounts. Here, 
we present a powerful classification scheme that exclusively uses the natural language text from 
organic users to provide a criterion for identifying accounts posting automated messages. Since the 
classifier operates on text alone, it is flexible and may be applied to any textual data beyond the 
Twittersphere. 

PACS numbers: 


I. INTRODUCTION 

Twitter has become a mainstream social outlet for the 
discussion of a myriad of topics through microblogging 
interactions. Members chiefly communicate via short 
text-based public messages restricted to 140 characters, 
called tweets. As Twitter has evolved from a simple 
microblogging social media interface into a mainstream 
source of communication for the discussion of current 
events, politics, consumer goods/services, it has become 
increasingly enticing for parties to gameify the system by 
creating automated software to send messages to organic 
(human) accounts as a means for personal gain and for 
influence manipulation mm- The results of sentiment 
and topical analyses can be skewed by robotic accounts 
that dilute legitimate public opinion by algorithmically 
generating vast amounts of inorganic content. Neverthe¬ 
less, data from Twitter is becoming a source of interest 
in public health and economic research in monitoring the 
spread of disease mm and gaining insight into public 
health trends [5]. 

In related work Em, researchers have built classifi¬ 
cation algorithms using metadata idiosyncratic to Twit¬ 
ter, including the number of followers, posting frequency, 
account age, number of user mentions/replies, username 
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length, and number of retweets. However, relying on 
metadata can be problematic: sophisticated spam algo¬ 
rithms now emulate the daily cycle of human activity and 
author borrowed content to appear human [6]. Anoth¬ 
er problematic spam tactic is the renting of accounts of 
legitimate users (called sponsored accounts), to introduce 
short bursts of spam and hide under the user’s organic 
metadata to mask the attack m- 

A content based classifier proposed by m measures 
the entropy between Twitter time intervals along with 
user meta data to classify Twitter accounts, and requires 
a comparable number of tweets (> 60) for adequate 
classification accuracy as our proposed method. Senti- 
Bot, another content based classifier m, utilizes latent 
Dirichlet allocation (LDA) for topical categorization 
combined with sentiment analysis techniques to classi¬ 
fy individuals as either bots or humans. We note that as 
these automated entities evolve their strategies, combi¬ 
nations of our proposed methods and studies previously 
mentioned may be required to achieve reasonable stan¬ 
dards for classification accuracy. Our method classifies 
accounts solely based upon their linguistic attributes and 
hence can easily be integrated into these other proposed 
strategies. 

We introduce a classification algorithm that operates 
using three linguistic attributes of a user’s text. The 
algorithm analyzes: 

1. the average URL count per tweet 

2. the average pairwise lexical dissimilarity between a 
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user’s tweets, and 

3. the word introduction rate decay parameter of the 
user for various proportions of time-ordered tweets 

We provide detailed descriptions of each attribute in 
the next section. We then test and validate our algorithm 
on 1 000 accounts which were hand coded as automated 
or human. 

We find that for organic users, these three attributes 
are densely clustered, but can vary greatly for automa¬ 
tons. We compute the average and standard deviation of 
each of these dimensions for various numbers of tweets 
from the human coded organic users in the dataset. We 
classify accounts by their distance from the averages from 
each of these attributes. The accuracy of the classifier 
increases with the number of tweets collected per user. 
Since this algorithm operates independently from user 
metadata, robotic accounts do not have the ability to 
adaptively conceal their identities by manipulating their 
user attributes algorithmically. Also, since the classifi¬ 
er is built from time ordered tweets, it can determine 
if a once legitimate user begins demonstrating dubious 
behavior and spam tactics. This allows for social media 
data-miners to dampen a noisy dataset by weeding out 
suspicious accounts and focus on purely organic tweets. 

II. DATA HANDLING 

A. Data-Collection 

We filtered a 1% sample of Twitter’s streaming API 
(the spritzer feed) for tweets containing geo-spatial meta¬ 
data spanning the months of April through July in 2014. 
Since roughly 1% of tweets provided GPS located spa¬ 
tial coordinates, our sample represents nearly all of the 
tweets from users who enable geotagging. This allows 
for much more complete coverage of each user’s account. 
From this sample, we collected all of the geo-tweets from 
the most active 1 000 users for classification as human or 
robot and call this the Geo-Tweet dataset. 


B. Social HoneyPots 

To place our classifier in the context of recent work, 
we applied our algorithm to another set of accounts col¬ 
lected from the Social HoneyPot Experiment m This 
work exacted a more elaborate approach to find auto¬ 
mated accounts on Twitter by creating a network of 
fake accounts (called Devils [12]) that would tweet about 
trending topics amongst themselves in order to tempt 
robotic interactions. The experiment was analyzed and 
compiled into a dataset containing the tweets of “legit¬ 
imate users” and those classified as “content polluters”. 
We note that the users in this dataset were not hand cod¬ 
ed. Accounts that followed the Devil honeypot accounts 
were deemed robots. Their organic users were compiled 
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FIG. 1: The feature distribution of the 1000 hand coded 
users are summarized with histograms and violin plots. These 
show the wide variation in automated features versus Organ¬ 
ics. Violin plots show the kernel density estimation of each 
distribution. Using the Organic features, automated entities 
are identified by exclusion. 

from a random sample of Twitter, and were only deemed 
organic because these accounts were not suspended by 
Twitter at the time. Hence the full HoneyPot dataset 
can only serve as an estimate of the capability of this 
classification scheme. 


C. Human Classification of Geo-Tweets 

Each of the 1 000 users were hand classified separately 
by two evaluators. All collected tweets from each user 
were reviewed until the evaluator noticed the presence 
of automation. If no subsample of tweets appeared to 
be algorithmically generated, the user was classified as 
human. The results were merged, and conflicting entries 
were resolved to produce a final list of user ids and cod¬ 
ings. See Figure 1 for histograms and violin plots sum¬ 
marizing the distributions of each user class. We note 
that any form of perceived automation was sufficient to 
deem the account as automated. See SI for samples of 
each of these types of tweets from each user class and a 
more thorough description of the annotation process. 

D. Types of Users 

We consider organic content, i.e. from human 
accounts, as those that have not tweeted in an algo¬ 
rithmic fashion. We focused on three distinct classes of 
automated tweeting: 

Robots: Tweets from these accounts draw on a 
strictly limited vocabulary. The messages follow a very 
structured pattern, many of which are in the form 
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of automated updates. Examples include Weather 
Condition Update Accounts, Police Scanner Update 
Accounts, Help Wanted Update Accounts, etc. 

Cyborgs: The most covert of the three, these 

automatons exhibit human-like behavior and messages 
through loosely structured, generic, automated messages 
and from borrowed content copied from other sources. 
Since many malicious cyborgs on Twitter try to market 
an idea or product, a high proportion of their tweets 
contain URLs, analogous to spam campaigns studied 
on Facebook m- Messages range from the backdoor 
advertising of goods and services m to those trying 
to influence social opinion or even censor political 
conversations m- These accounts act like puppets from 
a central algorithmic puppeteer to push their product 
on organic users while trying to appear like an organic 
user m ■ Since these accounts tend to borrow content, 
they have a much larger vocabulary in comparison to 
ordinary robots. Due to Twitter’s 140 character-per- 
tweet restriction, some of the borrowed content being 
posted must be truncated. A notable attribute of many 
cyborgs is the presence of incomplete messages followed 
by an ellipsis and a URL. Included in this category are 
‘malicious promoter’ accounts m that are radically 
promoting a business or an idea systematically. 

Human Spammers: These are legitimate accounts 
that abuse an algorithm to post a burst of almost indis¬ 
tinguishable tweets that may differ by a character in 
order to fool Twitter’s spam detection protocols. These 
messages are directed at a particular user, commonly for 
a follow request to attempt to increase their social reach 
and influence. 

Although we restrict our focus to the aforementioned 
classes, we did notice the presence of other subclass¬ 
es, which we have named “listers”, and “quoters”, that 
have both organic and automaton features. Listers are 
accounts that send their messages to large groups of indi¬ 
viduals at once. Quoters are dedicated accounts that are 
referencing distant passages from literature or song lyrics. 
Most of the tweets from these accounts are all encased in 
quotations. These accounts also separately tweet organic 
content. We classified these accounts as human because 
there was not sufficient evidence suggesting these behav¬ 
iors were indeed automated. 


Although we have classified each automaton into three 
distinct classes, the classifier is built more simply to 
detect and separate organic content from automated. To 
classify the tweets from a user, we measure three distinct 
linguistic attributes: 

1. Average Pairwise Tweet Dissimilarity, 

2. Word Introduction Rate Decay Parameter, 

3. Average number of URLs (hyperlinks) per tweet. 


B. Average Pairwise Tweet 
Dissimilarity 

Many algorithmically generated tweets contain similar 
structures with minor character replacements and long 
chains of common substrings. Purely organic accounts 
have tweets that are very dissimilar on average. The 
length of a tweet, £, is defined as the number of char¬ 
acters in the tweet and is denoted \t\. Each tweet is 
cleaned by truncating multiple whitespace characters and 
the metric is performed case insensitively. A sample of 
s tweets from a particular user is denoted T®. Given a 
pair of tweets from a particular user, +, tj £ T®, the pair¬ 
wise tweet dissimilarity, D(ti,tj ), is given by subtracting 
the length of the longest common subsequence of both 
tweets, \LCS(ti,tj)\ and then weighting by the sum of 
the lengths of both tweets: 


nr, , > _ IM + fel - 2 • |LOS(f„yi 

M + M 

The average tweet dissimilarity of user /i for sample size 
of s tweets is calculated as: 

V2 ) u^eTs 

For example, given the two tweets: 

(t\ ,£ 2 ) = (I love Twitter, I love to spam). Then \t\\ = 
|^ 2 1 = 14, LCS(ti,t 2 ) = 11 love 11 = 8 (including whites¬ 
paces) and we calculate the pairwise tweet dissimilarity 
as: 


D(t 1 ,t 2 ) 


14 + 14-2-8 
14 + 14 


12 

28 


3 
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III. METHODS 
A. Classification Algorithm 

The classifier, C, takes ordinal samples of tweets from 
each user, /i, of varying number, s, to determine if the 
user is a human posting strictly organic content or is 
algorithmically automating tweets: 

C : (i s —{0,1} = {Organic, Automaton}. 


C. Word Introduction Decay Rate 

Since social robots automate messages, they have 
a limited and crystalline vocabulary in comparison to 
organic accounts. Even cyborgs that mask their automa¬ 
tions with borrowed content cannot fully mimic the rate 
at which organic users introduce unique words into their 
text over time. The word introduction rate is a mea¬ 
sure of the number of unique word types introduced 
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Word Introduction User Features: 50 Tweets 



— Cyborgs 

— Robots 
- Spammers 


10 ° 10 1 10 2 10 3 

Rank 


FIG. 2: The rank-unique word gap distribution is plotted on 
a logscale for each user class. 

over time from a given sample of text H3- The rate at 
which unique words are introduced naturally decays over 
time, and is observably different between automated and 
organic text. By testing many random word shufflings 
of a text, we define m n as the average number of words 
between the n th and n + 1 st initial unique word type 
appearances. From m, the word introduction decay 
rate, a(n), is given as 

a(n) = 1 /mF cc n -7 for 7 > 0 . 

For each user, the scaling exponent of the word intro¬ 
duction decay rate, <a, is approximated by performing 
standard linear regression on the last third of the log- 
transformed tail of the average gap size distribution as a 
function of word introduction number, n PH- In figure |2] 
below, the log transformed rank-unique word gap distri¬ 
bution is given for each individual in the data set. Here 
the human population (green) is distinctly distributed in 
comparison to the automatons. 

D. Average URLs per Tweet 

Hyperlinks (URLs) help automatons spread spam and 
malware mmm ■ A high fraction of tweets from 
spammers tend to contain some type of URL in com¬ 
parison to organic individuals, making the average URLs 
per tweet a valuable attribute for bot classification algo¬ 
rithms nam ng. For each user, the average URL rate is 
measured by the total number of occurrences of the sub¬ 
string ‘http:’ within tweets, and then divided by the total 
number of tweets authored by the user in the sample of 


size s: 

s ^Occurrences of ‘http:’ 

^ url #Sampled Tweets 

E. Cross Validation Experiment 

We perform a standard 10-fold Cross Validation pro¬ 
cedure on the 2014 Geo-Tweet data set to measure the 
accuracy of using each linguistic feature for classifying 
Organic accounts. We divided individuals into 10 equal¬ 
ly sized groups. Then 10 trials are performed where 9 of 
the 10 groups are used to train the algorithm to classify 
the final group. 

During the Calibration phase, we measure each of 
the three features for every human coded account in 
the training set. We sequentially collect tweets from 
each user from a random starting position in time. We 
record the arithmetic mean and standard deviation of the 
Organic attributes to classify the remaining group. The 
classifier distinguishes Human from Automaton by using 
a varying threshold, n, from the average attribute value 
computed from the training set. For each attribute, we 
classify each user as an automaton if their feature falls 
further than n standard deviations away from the organic 
mean, for varying n. 

For each trial, the False Positives and True Positives 
for a varying window size, n, are recorded. To compare 
to other bot-detection strategies, we rate True Positives 
as the success at which the classifier identifies automa¬ 
tons by exclusion, and False Positives as humans that 
are incorrectly classified as automatons. The results 
of the trials for varying tweet sizes are averaged and 
visualized with a Receiver Operator Characteristic curve 
(ROC) (see Figure [3|. The accuracy of each experiment 
is measured as the area under the ROC, or AUC. To 
benchmark the classifier, a 10 -fold cross validation was 
also performed on the HoneyPot tweet-set which we 
describe in the following section. 


IV. RESULTS AND DISCUSSION 
A. Geo-Tweet Classification Validation 

The ROC curves for the Geo-Tweet 10 fold Cross Val¬ 
idation Experiment for varying tweet bins in Figure [3] 
show that the accuracy increases as a function of num¬ 
ber of tweets. 

Although the accuracy of the classifier increases with 
the number of collected tweets, we see in Figure [4] that 
within 50 tweets the accuracy of the average of 10 ran¬ 
dom trials is only slightly higher than a 500 tweet user 
sample. While this is very beneficial to our task (isolating 
humans), we note that larger samples see greater returns 
when one instead wants to isolate spammers, that tweet 
random bursts of automation. 
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▼ (FP,TP,N) = (0.08,0.84,2.25) 


100 Tweets, AUC = 0.95 
A (FP,TP,N) = (0.05,0.88,2.7) 

- 200 Tweets, AUC = 0.95 

■ (FP,TP,N) = (0.07,0.89,2.2) 

300 Tweets, AUC = 0.96 

• (FP,TP,N) = (0.07,0.9,2.3) 
- 400 Tweets, AUC = 0.97 

♦ (FP,TP,N) = (0.06,0.91,2.5) 
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FIG. 3: The receiver operator characteristic curve from the 
10-fold Cross Validation Experiment performed on the Geo 
Tweets collected from April through July 2014. The True 
Positive (TP), False Positive (FP), and thresholds, V, are 
averaged across the 10 trials. The accuracies are approximat¬ 
ed by the AUCs, which we compute using the trapezoid rule. 
The points depict the best experimental model thresholding 
window (N). 



FIG. 4: Accuracy, computed as the AUC is plotted as a func¬ 
tion of number of tweets, ranging from 25 to 500. The average 
True Positive and False Positive Rates over 10 trials is given 
on twin axes with error bars drawn using the standard error. 


B. HoneyPot External Validation 

The classifier was tested on the Social Honeypot 
Twitter-bot dataset provided by DU- Results are visu¬ 
alized with a ROC curve in Figure [5j The averaged 
optimal threshold for the full English user dataset (blue 
curve) had a high true positive rate (correctly classified 
automatons: 86%), but also had a large false positive 
rate (misclassified humans: 22%). 

The Honeypot Dataset relied on Twitter’s spam detec¬ 
tion protocols to label their randomly collected “legiti¬ 
mate users”. Some forms of automation (weather-bots, 
help-wanted bots) are permitted by Twitter. Other 
cyborgs that are posting borrowed organic content can 
fool Twitter’s automation criterion. This ill formation of 
the training set greatly reduces the ability of the classifier 
to distinguish humans from automatons, since the clas¬ 
sifier gets the wrong information about what constitutes 
a human. To see this, a random sample of 1 000 English 
Honeypot users was hand-coded to mirror the previous 
experiment. On this smaller sample (black curve in Fig¬ 
ure 4), the averaged optimal threshold accuracy increased 
to 96%. 



False Positive Rate 


FIG. 5: Honey Pot Data Set, 10 fold Cross Validation Perfor¬ 
mance for users with 200 tweets. The black curve represents 
the 1 000 hand coded HoneyPot users, while the blue curve is 
the entire English Honeypot dataset. The accuracy increases 
from 84% to 96%. 


C. Calibrated Classifier Performance 

We created the thresholding window of final calibrat¬ 
ed classifier using the results from the calibration exper¬ 
iment. We average the optimal parameters from the 10 
fold cross validation on the Geo-Tweet dataset from each 


of the 10 calibration trials for tweet bins ranging from 25 
to 500 in increments of 25 tweets. We also average and 
record the optimal parameter windows, n opt and their 
standard deviations, cr opt . The standard deviations serve 
as a tuning parameter to increase the sensitivity of the 
classifier, by increasing the feature cutoff window (n). 
The results from applying the calibrated classifier to the 
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400 Tweets: TP = 90.32% FP = 4.79% 



FIG. 6: Calibrated Classifier Performance on 1 000 User Geo- 
Tweet Dataset. Correctly classified humans (True Negative), 
are coded in Green, while correctly identified automatons 
(True Positives) are coded in red. The 400 tweet average opti¬ 
mal thresholds from the cross validation experiment designate 
the thresholding for each feature. The black lines demon¬ 
strates each feature cutoff. 

full set of 1 000 users, using 400 tweet bags is given in Fig¬ 
ure [6] The feature cutoff window (black lines) estimates 
if the user’s content is organic or automated. Human fea¬ 
ture sets (True Negatives: 716) are densely distributed 
with a 4.79% False Positive Rate (i.e., humans classified 
as robots). The classifier accurately classified 90.32% 
of the automated accounts and 95.21% of the Organic 
accounts. See Figure SI for a cross sectional comparison 
of each feature set. We note that future work may apply 
different methods in statistical classification to optimize 
these feature sets, and that using these simple cutoffs 
already leads to a high level of accuracy. 

V. CONCLUSION 

Using a flexible and transparent classification scheme, 
we have demonstrated the potential of using linguistic 
features as a means of classifying automated activity on 
Twitter. Since these features do not use the metada¬ 
ta provided by Twitter, our classification scheme may 
be applicable outside of the Twittersphere. Future work 
can extend this analysis multilingually and incorporate 
additional feature sets with an analogous classification 
scheme. URL content can also be more deeply analyzed 
to identify organic versus SPAM related hyperlinks. 

We note the potential for future research to investigate 
and to distinguish between each sub-class of automa¬ 


ton. We formed our taxonomy according to the differ¬ 
ent modes of text production. Our efforts were primar¬ 
ily focused in separating any form of automation from 
organic,human content. In doing so we recognized three 
distinct classes of these types of automated accounts. 
However, boundary cases (e.g. cyborg-spammers, robot- 
spammers, robotic-cyborgs, etc.) along with other poten¬ 
tial aforementioned subclasses (e.g. listers, quoters, etc.) 
can limit the prowess of our current classification scheme 
tailored towards these subclasses. We have shown that 
human content is distinctly different from these forms of 
automation, and that for a binary classification of auto¬ 
mated or human, these features have a very reasonable 
performance with our proposed algorithm. 

Our study distinguishes itself by focusing on auto¬ 
mated behavior that is tolerated by Twitter, since 
both types of inorganic content can skew the results of 
sociolinguistic analyses. This is particularly important, 
since Twitter has become a possible outlet for health 
economics [4] research including monitoring patient 
satisfaction and modeling disease spread j3j [24 . Mon¬ 
itoring excessive social media marketing of electronic 
nicotine delivery systems (also known as e-cigarettes), 
discussed in [25], [26], makes classifying organic and 
automated activity relevant for research that can benefit 
policy-makers regarding public health agendas. Isolating 
organic content on Twitter can help dampen noisy 
data-sets and is pertinent for research involving social 
media data and other linguistic data sources where a 
mixture of humans and automatons exist. 

In health care, a cardinal problem with the use of elec¬ 
tronic medical records is their lack of interoperability. 
This is compounded by a lack of standardization and 
use of data dictionaries which results in a lack of preci¬ 
sion concerning our ability to collate signs, symptoms, 
and diagnoses. The use of millions or billions of tweets 
concerning a given symptom or diagnosis might help to 
improve that precision. But it would be a major set¬ 
back if the insertion of data tweeted from automatons 
would obscure useful interpretation of such data. We 
hope that the approaches we have outlined in the present 
manuscript will help alleviate such problems. 
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Supplementary materials 


SI: Cross Sectional Classifier Performance 


FIG. SI Calibrated Classifier Performance on 1 000 User Geo Tweet Dataset. Correctly classified humans (True 
Negatives), are coded in Green, while correctly identified automatons (True Positives) are coded in red. The 400 
tweet average optimal thresholds from the cross validation experiment designate the thresholding for each feature. 
The black lines demonstrate each feature cutoff. 
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S2: Model Comparisons 


Receiver Operator Characteristic (ROC) Curves of the performance of each individual feature are given in FIG. 
S2 below. We see each feature set performs comparably with accuracies (measured as AUC) ranging from 80% to 
91% depending on the number of tweets compiled in the analysis. Combinations of each metric greatly increases the 
classification accuracy, the apparent most accurate model uses all three features. However, it is notable that combi¬ 
nations of two of these features perform strongly in comparison. It is also notable that the word introduction decay 
parameter coupled with the average URL rate performs as well as the Dissimilarity-URL model. The Dissimilarity 
metric requires determining the Longest Common Substring between many sets of tweets which is computationally 
expensive compared to analytically calculating the Word Introduction Decay Parameter. 
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S3: Word Introduction Decay Parameter 


Here, we expand upon our description of the Word Introduction Decay Parameter. This parameter is based upon 
random word shufflings of a text, but is computed via the analytic formula given in Eq 8. of [16]. To determine 
the decay rate parameter, we: (1) compute the word introduction rate as a function of word number, n, and (2) 
regress in log-log space for a power law decay rate parameter measuring in the final third for the tail, where the decay 
rate assumes the form of a power law. While this heuristic is crude and could certainly be refined to more precisely 
measure the power law region, which can vary with corpus size, the tightness of organic-user clustering afforded by 
this parameter, coupled with its computationally cheap cost when compared to the pairwise tweet dissimilarity metric 
affords us great power for bot discrimination. 

In Figure S3 below the unique word introduction gaps are plotted in log-space as a function of unique word 
introduction number (rank) for each individual in our data set for various numbers of tweets. We see the distribution 
growing with the number of tweets. However, at each resolution, the human class is very distinctly distributed in 
comparison to each form of automation. Even within 25 tweets, the human clustering is visually apparent versus their 
automated counterparts. 
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In Figure S4 below, we visualize the stability of this parameter between an individual’s set of tweets. The tweets 
from each account in our data set were resampled 100 times to recompute the word introduction decay parameter for 
25 tweets (top) and 400 tweets (bottom). The standard deviation between each account’s 100 decay parameters is 
given in the histogram. The average standard deviation across all individuals of each set, /i, is given in the title of 
each histogram. Notably, the humans have very little deviation (i.e. within the ’window of forgiveness’) for both sets 
of tweets. Automated classes, in particular spammers, can vary quite wildly depending on the sample of tweets that 
are analyzed. In particular, spammers look similar to (and usually are) humans and if the spamming event is not 
captured in the sampled data they will be misclassified. This decay parameter for human text is robust for varying 
sets of tweets and is quite distinguishable from automated accounts. 


Word Introduction Decay Standard Deviation Distribution: 25 Tweets 

Humans: ix = 0.02, N = 752 Cyborgs: y = 0.06, N = 91 Robots: /i = 0.07, N = 54 Spammers: n = 0.25, N = 103 
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S4: Human Annotation of Twitter Accounts 


In this section we describe the annotation process for 
classifying user accounts. Each of the one-thousand 
accounts were separately classified by two evaluators. All 
of the collected tweets from each account were assessed 
until the presence of automation was uncovered. An 
account was coded as ‘human’ if no automated posting 
presence was detected (i.e. an algorithm posting on the 
individual’s behalf). The inter-rater reliability is sum¬ 
marized in the table below by listing the classification 
discrepancies between account classes. For each class, 
the counts of the type of rating are displayed. For exam¬ 
ple, the Human class had a total of 6 account discrep¬ 
ancies which is composed of 12 different scores (6 from 
each rater) - 5 human codings, 3 robot, 0 cyborg, and 4 
spammers. 

The reliability between raters was favorable (91.49%) 
for the entire dataset. The largest source of discrep¬ 
ancies were from the ‘Robot’ and ‘Spammer’ classes. 
The ’Spammer’ class was most confused with ’Humans’- 
which is intuitive because many of these individuals were 
humans that utilized a algorithm to SPAM a particu¬ 
lar message. ‘Robots’ were commonly confused with 
’cyborgs’. This is most likely due to boundary cases 
regarding both classes. The boundaries between these 
classes can at time be ambiguous. We classified cyborgs 
as automatons that were posting ‘borrowed content‘ from 
another source or an account that used human assist¬ 
ed automation, i.e. a human that could be overseeing 
an automated account. Robots were defined as strictly 
posting structured automated messages in the forms of 
updates. Perhaps future work can work to sub classify 
different types of robots and cyborgs to investigate the 
ecology of these automatons. 


TABLE I: Annotation Discrepancies of Twitter Accounts 


Class 

Discrepancies 

Totals 


Human 

Robot 

Cyborg Spammer 


Human 

5 

3 

0 

4 

0.79% 

Cyborg 

2 

9 

6 

1 

9.89% 

Robot 

0 

31 

32 

1 

60.38% 

Spammer 

31 

4 

4 

37 

36.89% 

All 

6 

9 

32 

38 

8.51% 


Each discrepancy was revisited by both annotators and 
discussed until a class was determined. For extreme 
boundary cases, the account ID was searched via the 
hyperlink: https : //twitter. com/intent/user?user_ 
id=###, This helped observe other user features (screen 
name, description, etc.) to make a better decision about 
the user. This was especially helpful for identifying pro¬ 
motional accounts or news sources. 

Screen shots of particular accounts are given below 
to help describe the annotation process. Each anno¬ 
tator scrolled through a terminal interface containing 
each individual’s tweets. Scrolling through ’human‘ 


text appears un-ordered and chaotic with very little 
structure. Automated accounts have very structured 
messages, hence these patterns become very apparent in 
comparison to human accounts. 


Cyborg Account Example: A canonical cyborg’s 
tweets are given below. This particular automaton is a 
news promotional account that is tweeting links to arti¬ 
cles. Notice the description tailors off when it reaches the 
character limit and shows this with an ellipses (...) next 
to a URL. 

M>_ userTweets - more - 156x26_ 

Miley's Meadowlands show is surreal, lacks substance: review: Say what you will about Miley Cyrus, but she knows... http://t.co/CCQI3P7LL0 

I Bushwick warehouse owner invites tenants to 'join the gentrification': Yeah, it's gentrification - get over it- http://t.co/axMKzW5TYo 

Yankees Insider: No Mo? No problem for Diamond Dave: In his first opportunity as the full-time closer for the... http://t.co/VcrHPFOiyp 
Phew! Yanks avoid sweep as Ichiro and Solarte power Bombers to first victory: Breathe easy, Yankees fans. The... http://t.co/k3Qv30yD21 
■If you don't like it, sit your a— on a couch and eat a donut': CrossFit mom whose pregnant weightlifting photos... http://t.co/PHz83m5YgN 
Bronx Urban Cowboy back in the saddle: HE'S A real urban cowboy. Bronx-born rodeo star AJ Silver has lassoed and... http://t.co/93jAT4QnFd 
Wake-Up Call: De Blasio and Cuomo spat over speed enforcement cameras: Good morning and welcome to Wake-Up Call,... http://t.co/UwQqQqmQmA 
Cabbies' $1.4M health care fund remains frozen by city lawyers: A long-sought health care and disability fund for... http://t.co/yT4TAeqPhC 
J Judge tosses out Brooklyn gun seizure over cop's false testimony: The decision - which will likely result in the... http://t.co/6yIctqdXzE 
From Winter's Bite to Spring's: New Yorkers took advantage of the warmer weather in Hudson River Park in New York. http://t.co/VphjlR7IpW 

Sen. Charles Schumer could help to save Cit Bike: A commuter benefit introduced by Sen. Charles Schumer would..? http://t.co/z23ltjyKHA 
B'klyn Housing Court can't find a home: Brooklyn Housing Court is looking for a new home. The court has outgrown... http://t.co/axllFtlzP4 
It's Still Bad for the Long-Term Unemployed: The evidence remains that the so-so recovery is not enough to help... http://t.co/AX6cUmfoCK 
Most Details in Jobs Report Are Positive: The trend toward a rising labor force in 2014 suggests that America is... http://t.co/wIlTw9eMGN 
:BUBBA BEWARE! Bill Clinton asked aides to investigate aliens while in charge of the free world - 'We're not alone... http://t.co/Tetxao8AjM 

Panel looks to shelve big-spending library director: The Queens Library Board retreated into a private meeting... http://t.co/bRNhbmqFsP 
Derek Jeter avoids pesky adult fan to give souvenir ball to young girl: When Derek Jeter tells you to get out of... http://t.co/89msl7GqzF 
Raissman: Time to lower Boomer on pompous Pope!: For sports radio Gasbags, unintended consequences are a good... http://t.co/fh9xxJuF2K 
Phew! Yanks avoid sweep as Ichiro and Solarte power Bombers to first victory: Breathe easy, Yankees fans. The... http://t.co/gTMNbIAlhD 

Lupica: After Fort^Hood, time to invest in the mental health of our vets: The firs/reports out of Fort Hood... http://t.co/M7S6eeYUN0 
Gov. Cuomo wants to reform Port Authority: Gov. Cuomo wants to reform the scandal-tainted Port Authority of New... http://t.co/5Lid7sTHlQ 

Robot Account Example: Robots tweet generically 
structured messages, usually as a form of update. These 
automatons have a very limited vocabulary and in gen¬ 
eral only change a few characters per tweet. This robot 
(below) is an example of a weather update bot that is 
tweeting statistics about the weather at regular intervals. 


• • 

• 




userTweets 

— more — 156x26 

Temp: 

82. e! 


| Humidity: 53% | 

Wind: N @ 1.6 mph | 

Barometer: 30.01 in | 

Dewpoint: 63.3Sjl^F 

Temp: 

78.3j 


| Humidity: 56% | 

Wind: N @ 1.6 mph j 

Barometer: 30.01 in j 

Dewpoint: 61.35fflp 

Temp: 

61.3j 

2 F 

| Humidity: 93% | 

Wind: - @ 0.0 mph 

| Barometer: 30.03 in 

| Dewpoint: 59^fflsHF 

Temp: 

81. 0 ! 


| Humidity: 67% | 

Wind: E @ 2.2 mph | 

Barometer: 30.06 in | 

Dewpoint: 69.03^8^ 

Temp: 

86. ej 


| Humidity: 50% | 

Wind: SSE @ 1.6 mph 

| Barometer: 30.06 in 

| Dewpoint: 65^®^ 

Temp: 

88.0! 


j Humidity: 47% | 

Wind: W @ 1.6 mph | 

Barometer: 30.04 in | 

Dewpoint: 65 . 3 ^^h£ 

Temp: 

91.e! 


| Humidity: 38% | 

Wind: SE @ 0.7 mph | 

Barometer: 29.98 in 

| Dewpoint: 62.0^2 

Temp: 

70.3! 

S F 

| Humidity: 84% | 

Wind: - @ 0.0 mph 

| Barometer: 30.05 in 

| Dewpoint: 65.3gtf F 

Temp: 

66 . e! 


j Humidity: 93% | 

Wind: - @0.0 mph 

j Barometer: 30.05 in 

j Dewpoint: 64.0^ F 

Temp: 

66.0! 


| Humidity: 94% | 

Wind: - @0.0 mph 

j Barometer: 30.06 in 

| Dewpoint: 64.3gS F 

Temp: 

66.0! 


j Humidity: 94% | 

Wind: - @0.0 mph 

| Barometer: 30.05 in 

j Dewpoint: 64 . 3 ^X F 

Temp: 

92. 3 ! 


j Humidity: 42% | 

Wind: SSE @ 2.2 mph 

j Barometer: 30.07 in 

| Dewpoint: 66.0SJ F 

Temp: 

92.3! 


| Humidity: 42% | 

Wind: SSE @ 3.1 mph 

j Barometer: 30.08 in 

j Dewpoint: 66.0SJ F 

Temp: 

69. 3 ! 


j Humidity: 99% | 

Wind: - @0.0 mph 

j Barometer: 30.18 in 

j Dewpoint: 69.0^ F 

Temp: 

91.0j 


| Humidity: 49% | 

Wind: SSE @ 3.1 mph 

j Barometer: 30.18 in 

| Dewpoint: 69. 3 g |f 

Temp: 

91.0j 


| Humidity: 49% | 

Wind: SE @ 1.6 mph | 

Barometer: 30.17 in 

Dewpoint: 69.3^tt F 

Temp: 

73.0j 


j Humidity: 94% | 

Wind: SSE @ 0.7 mph 

| Barometer: 30.14 in 

| Dewpoint: 71 .Jf 

Temp: 

73.0! 


| Humidity: 94% | 

Wind: SE @ 0.0 mph | 

Barometer: 30.14 in 

Dewpoint: 71.3^ F 

Temp: 

81.0! 


j Humidity: 80% | 

Wind: SSE @ 3.1 mph 

| Barometer: 30.20 in 

| Dewpoint: 74.3® Jf 

Temp: 

88 . 3 j 


j Humidity: 59% | 

Wind: SE @ 2.2 mph | 

Barometer: 30.19 in 

Dewpoint: 72.3gffl F 

Temp: 

91.0! 


| Humidity: 52% | 

Wind: SE @ 3.1 mph | 

Barometer: 30.15 in 

Dewpoint: 71.0^ 

Temp: 

69. 3 ! 


| Humidity: 87% | 

Wind: - @0.0 mph 

| Barometer: 30.12 in 

| Dewpoint: 65.3® !f 

Temp: 

70.3! 


| Humidity: 90% | 

Wind: - @0.0 mph 

j Barometer: 30.12 in 

| Dewpoint: 67.3® 

Temp: 

73.0j 

2 F 

| Humidity: 87% | 

Wind: SE @ 1.6 mph | 

Barometer: 30.11 in 

Dewpoint: 69.0^^HF 

Temp: 

81. 0 ! 


j Humidity: 80% | 

Wind: SE @ 3.8 mph | 

Barometer: 30.16 in 

Dewpoint: 74.3 ®Sf 


Spammer Account Example: Tweets from a spam¬ 
ming human account are given below. This individual 
has utilized an algorithm to tweet at a musical celebri¬ 
ty. Many of these spam algorithms try to fool Twitter’s 
detector by including a different number or symbol at the 
end of the tweet. 


userTweets — more — 156x26 


@KeatonStromberg I cai 

i't wait t 

0 see your new 

cover ! 

I love you V#KeatonNewCover 96 

@KeatonStromberg I cai 

i't wait t 

0 see your new 

cover ! 

I love you V#KeatonNewCover 97 

@KeatonStromberg I cai 

i't wait t 

0 see your new 

cover ! 

I love you V#KeatonNewCover 98 

@KeatonStromberg I cai 

i't wait ti 

0 see your new 

cover ! 

I love you V#KeatonNewCover 99 

@KeatonStromberg I cai 

i't wait t 

0 see your new 

cover ! 

I love you V#KeatonNewCover 100 


@KeatonStromberg who else is excited 
@KeatonStromberg who else is excited 
@KeatonStromberg who else is excited 
@KeatonStromberg who else is excited 
@KeatonStromberg who else is excited 
@KeatonStromberg who else is excited 
@KeatonStromberg who else is excited 
@KeatonStromberg who else is excited 
@KeatonStromberg I tell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 
@KeatonStromberg I fell in love all 


for #KeatonNewCover ? I love you 
for #KeatonNewCover ? I love you 
for #KeatonNewCover ? I love you 
for #KeatonNewCover ? I love you 
for #KeatonNewCover ? I love you 
for #KeatonNewCover ? I love you 
for #KeatonNewCover ? I love you 
for #KeatonNewCover ? I love you 
over again when I saw “ 

over again when I saw 
over again when I saw 
over again when I saw 
over again when I saw 
over again when I saw 
over again when I saw 
over again when I saw 



over again when I saw this 
over again when I saw this 
over again when I saw this 
over again when I saw this 


this ^#KeatonNewCover https://t.co/4dxJla9TMv 
this ^#KeatonNewCover https://t.co/4dxJla9TMv 2 
this V#KeatonNewCover https://t.co/4dxJla9TMv 3 
this * https://t.co/4dxJla9TMv #KeatonNewCover 4 
this * https://t.co/4dxJla9THv #KeatonNewCover 5 
this » https://t.co/4dxJla9TMv #KeatonNewCover 6 
this ■» https://t.co/4dxJla9TMv #KeatonNewCover 7 
https://t.co/4dxJla9TMv #KeatonNewCover 6 
https://t.co/4dxJla9TMv #KeatonNewCover 9 
https://t.co/4dxJla9TMv #KeatonNewCover 10 
https://t.co/4dxJla9TMv #KeatonNewCover 11 
https://t.co/4dxJla9TMv #KeatonNewCover 12 
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S5: Mixed User Sample Tweets 


table ii: Robot Sample Tweets 


No. 

Tweet 

1 

#SuspiciousPerson 3955 W D JUDGE DR 32808 (7/26 22:19) #Orlando #MercyDrive 

2 

#AccidentWithRoadBlockage LEEVISTA BV &amp; S SEMORAN BV N/A (6/24 18:07) #Orlando #CentralBusinessDistrict 

3 

@USER the 1st mention of #sunnysmiles appears on your TL. Now is Trending Topic in United States! #trndnl 

4 

TRAFFIC STOP at SE 181ST AVE / SE PINE ST, GRESHAM, OR [Gresham Police #PG14000039852] 23:58 #pdx911 

5 

A 2002 Ford Ranger was just scanned near Cleveland, TN 37311 URL #myvinny ^startup #buyacar 

6 

Visiting #SantaCruz, ^California? Check out this great new app for news, weather, hotels, and food here! URL 

7 

Trend Alert: #heatJustinBieber. More trends at URL #trndnl URL 

8 

Temp: 76.5F — Humidity: 70% — Wind: — @USER 0.0 mph — Barometer: 30.01 in — Dewpoint: 66.OF 

9 

On Sunday 4, #WinUgly was Trending Topic in Pittsburgh for 10 hours: URL #trndnl 

10 

Wind 0.0 mph —. Barometer 1016.0 mb, Rising slowly. Temperature 67.6 F. Rain today 0.46 in. Humidity 96% 


table ill: Cyborg Sample Tweets 


No. 

Tweet 

1 

Indianapolis, IN suburb Family practice physi... - Soliant Health: (^Indianapolis, IN) URL #FamilyPractice #Job 

2 

Barnabas Health: Patient Care Associate (#LongBranch, NJ) URL #Nursing #Job #Jobs #TweetMyJobs 

3 

Overlake offers a low-cost way to check lungs for cancer early: Doctors at Overlake say they?re tired of waiting... URL 

4 

#TweetMyJobs #Nursing #Job alert: Opening — Accountable Healthcare Staffing — #Glendale, AZ URL #Jobs 

5 

Soliant Health #IT #Job: Cerner Jobs - Cerner Analyst - San Diego, CA ( #SanDiego , CA) URL #Jobs #TweetMyJobs 

6 

Tyco #Marketing #Job: Digital Marketing Specialist ( #Monroe , NC) URL #Jobs #TweetMyJobs 

7 

@USER Timing is everything when announcing a breakup URL 

8 

Southwest flights briefly diverted to DFW Airport on Friday: A Southwest Airlines plane experiencing mechanical... URL 

9 

Fort Carson To Welcome Home About 225 Soldiers: FORT CARSON, Colo. (AP) ? Fort Carson will welcome home about 225... URL 

10 

Joint venture secures $97M in financing for two Boston hotels: Commonwealth Ventures and Ares Management have... URL 


table IV: Spammer Sample Tweets 


No. 

Tweet 

1 

#CallMeCam n#CallMeCam n @USER n nlf Cameron called me it’ll seriously make my day I love you please call me! 100 

2 

#CallMeCam n#CallMeCam n @USER n nlf Cameron called me it’ll seriously make my day I love you please call me! 321 

3 

#CallMeCam n^CallMeCam n @USER n nlf Cameron called me it’ll seriously make my day I love you please call me! 167 

4 

S/o to @USER thanks for the support. Check out my music @USER URL I promise u won’t be disappointed. 

5 

S/o to @USER destiiny thanks for the support. Check out my music @USER URL I promise u won’t be disappointed. 

6 

S/o to @USER thanks for the support. Check out my music @USER URL I promise u won’t be disappointed. 

7 

nAshton Irwin from 5SOS n nMy birthday is in 11 days, nAnd it would be an amazing gift, nlf you could follow me. Ily n @USER n nX3126 

8 

nAshton Irwin from 5SOS n nMy birthday is today, nAnd it would be an amazing gift, nlf you could follow me. Ily n @USER n nX5408 

9 

nAshton Irwin from 5SOS n nMy birthday is in 22 days, nAnd it would be an amazing gift, nlf you could follow me. Ily n @USER n nX765 

10 

nAshton Irwin from 5SOS n nMy birthday is in 8 days, nAnd it would be an amazing gift, nlf you could follow me. Ily n @USER n nX3422 
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TABLE V: Human Sample Tweets 


No. 

Tweet 

1 

I’ll marry whoever comes thru with some food 

2 

Ewwww them seats #BlackInkCrew 

3 

@USER if you only knew ???? i like you 

4 

Really wish he wasn’t so **** busy ?? 

5 

My son’s name is Gabriel. 

6 

guess I need to get up and get ready then 

7 

Grandma stayed on me bout not wearing socks in her house aint nobody got time for that 

8 

Thank you for reading. ?? 

9 

@USER: If only I knew.. ??? 

10 

WHY ARE YOU SO HOT URL 





